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ABSTRACT 

The impact of the use of data arising from balanced 
incomplete block (BIB) spiralled designs on the chi-square 
goodness-of-f it test in factor analysis is considered. Data from BIB 
designs posses a unique pattern of missing data that can be 
characterized as missing completely at random (MCAR) . Standard 
approaches to factor analyzing such data rest on forming pairwise 
available case (PAC) correlation matrices. Developments in 
statistical theory for missing data show that PAC correlation 
matrices may not satisfy Wishart distribution assumptions underlying 
factor analysis, this impacting tests of model fit. A new approach 
for handling missing data in structural equation modeling advocated 
by B. Muthen, D. Kaplan, and M. Hollis (1987) is proposed as a 
possible solution to these problems. The new approach is compared to 
the standard PAC approach in a Monte Carlo simulation framework. 
Simulation results show that tests of goodness-of -f it are very 
sensitive to PAC approaches even when data are MCAR, as is the case 
for BIB designs. The new approach outperforms the PAC approach for 
continuous variables and is comparatively much better for dichotomous 
variables. One table and one figure illustrate the discussion. 
(SLD) 
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The Impact of 
BIB-Spiralling induced 
Missing Data Patterns on 
Goodness-of-Fit Tests in 
Factor Analysis 



David Kaplan 
Department of Educational Studies 
University of Delaware 
Newark, DE 



In studies of adult literacy it is often desirable to assess the 
underlying dimensions of literacy performance. However, tradi- 
tional applications of such methods as factor analysis ignore the 
fact that assessments of adult literacy performance utilize a com- 
plex matrix sampling technique referred i ^ as balanced incomplete 
spiraling. Such complex matrix sampling can have a deleterious 
impact on the results of factor analysis. This paper considers the 
impact of the use of data arising from balanced incomplete block 
(BIB) spiralled designs on the chi-square goodness-of-fit test in 
factor analysis. Specifically, data arising from BIB designs possess 
a unique pattern of missing data that can be characterized as 
missing completely at random (MCAR). Standard approaches to 
factor analyzing such data rest on forming pairwise available case 
(PAC) correlation matrices. Developments in statistical theory for 
missing data show thai PAC correlation matrices may not satisfy 
Wishart distribution assumptions underlying factor analysis, thus 
impacting tests of model fit, A new approach for handling missing 
data instructuralequationmodelingadvocatedbyMuthen, Kaplan, 
and Mollis (1987) is proposed as a possible solution to these 
problems. This study compares the new approach to the standard 
PA C approach in a Mome Carlo simulation framework. Results of 
these simulations show that tests ofgoodness-offit are very sensi- 
tive to PAC approaches even when data are MCAR, as is the case 
for BIB designs. The new approach is shown to outperform the PAC 
approach for continuous variables and is comparatively much 
better for dichotomous variables. 
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7. Introduction 



In large scale national assessments such as the Young Adult Literacy 
Survey sponsored by the National Assessment of Educational Progress 
(NAEP) it is not always feasible or desirable for every test item to be 
administered to every respondent, Nevertheless, it is necessary to ensure 
a broad and representative coverage of the content of the assessment. One 
way in which such representation is accomplished is through a variant of 
matrix sampling referred to as balanced incomplete block (BIB) spiralling 
(Beaton, Johnson, & Ferris, 1987), In essence, a total pool of items is 
divided into a smaller number of blocks of tasks. The blocks are then 
assigned to booklets in such a way as to pair each block with every other 
blockin some booklet The administration of booklets is cycled (spiralled) 
so that each booklet is completed by a random sample of respondents. For 
example. Figure 1 shows the assignment of blocks to booklets for the 
NAEP Young Adult Literacy Survey (Kinch & Jungeblut, 1986) where 
each block contains approximately fifteen literacy items. Desirable char- 
acteristics of the BIB sph^alling approach are that (a) each block appears 
with the same frequency, (b) positional effects are controlled due to the 
fact that each block appears once in each of three possible positions, and 
(3) every pair of blocks appears with exactiy one booklet (Kirsch & 
Jungeblut, 1986, pg U). 

Figure 1 . Assignm^t of blocks to booklets for the NAEP Young 
Aduit Literacy Assessment 
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Given that it is of interest to study the underlying dimensions of adult 
literacy performance, more information must be obtained about the effects 
of BIB spiralled designs on results of factor analysis- One feature of data 
arising from a BIB design is that the resulting correlation matrix of BIB 
spiralled items possesses a particular pattern of missing data. This missing 
data pattern arises fiom the fact that estimates of die correlations between 



NATIONAL CtNTlK ON ADULT UTftACY 



1 



items within a block are based on a larger number of respondents i 

compared to the correlations of items within different blocks. In the case | 

of the NAEP reading data (Zwick, 1987), the ratio of these sample sizes | 

is about 9 to 1 while for the NAEP Young Adult Literacy Survey (Kirsch i 

& Jungeblut, 1986) the ratio is about 3 to L As Zwick (1987) has noted, | 

the random sampling nature of BIB spiralled data does not guarantee that | 

dimensionality assessment is unafiRected by this pattern of massing data. In | 

fact, it is known that pairwise available-case (PAC) methods of calcula^ J 

ing correlation matrices can yield problems for factor analysis even if the § 

data are missing completely at random (Little & Rubin, 1987). | 

To study the effects of BIB spiralled data on dimensionality assess- i 

ment Zwick (1987), among other things, simulated unidimensional com- | 

plete data and BIB spiralled data and applied principle component | 

analysis and full information factor analysis (Bock, Gibbons, & Muraki, | 

1985) to these simulated data sets. When descriptive statistics were | 

examined, Zwick' s results showed that BIB spiralled daca had little impact | 

on the iiecovery of the underlying dimensionality when compared to | 

complete data. However, an examination of the chi-square goodness-of- | 

fit statistics rejected the hypothesis of a single underlying factor for both | 

a complete data case and the BIB data case despite the fact that this was | 

the model that generated the data. Zwick notes that although both cases j 

lead to the rejection of unidimensionality, the value of the test statistic for || 

the BIB case was approximately two-thirds the value of the complete data | 

case. Zwick attributes this difference to the number of respondents per | 

item wherein she finds a similar ratio, 1 

i 

While it is true that sample size affects the chi-square test, two | 

alternative explanations of Zwick' s are also plausible. First, Zwick | 

generated only one finite sample replication of BIB and complete data. | 
With one sample, the effects of BIB spiralling on the empirical distribu- 
tion of the chi-square goodness-of-fit test cannot be adequately assessed. 
Second, with respect to the BIB case, the goodness-of-fit test may be 
sensitive to PAC data. This counter-explanation is plausible since PAC 
covariance matrices do not possess the correct statistical properties 
assumed by maximum likelihood (or asymptotically equivalent) theory 

(Brown, 1983). | 

Given these concerns, the purpose of this paper is to more fiiUy explore | 

the effects of BIB spiralled data on goodness-of-fit in an expanded Monte | 

Carlo simulation framework. Moreover, a new approach to assessing the J 

factor structure of BIB type data is presented which is hypothesized to | 

mitigate some of tiie problems associated with analyzing PAC covariance | 

matrices. Specifically, the application of an estimator developed by | 

Muthen, Kaplan, and Hollis (1987) in the context of structural equation | 
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modeling with missing data is proposed and compared to the standard 
approach of analyzing PAC correlation matrices in a Monte Carlo simu- 
lation context The organization of this paper is as follows: 



• Section 2 outlines the statistical problems associated with 
missing data, patterns of the BIB kind. 

• Section 3 presents the new estimator proposed by Muthen 
et al. (1987). 

• Section 4 presents the design of a small Monte Carlo 
study comparing the new estimator against the standard 
approach to analyzing BIB data. 

• Section 5 presents the results. 

• Section 6 concludes. 



2. Problems with BIB Spiralled Induced 
Missing Data Patterns 



To motivate the proposed estimator in Section 3 below, it is useful to 
review the known problems of analyzing PAC covariance matrices. 
Following Litde and Rubin (1987, pg. 42) consider the computation of 
j]^th p\Q covariance between two variable Yj and Yj^ based on cases for 
which yj and yj^ are observed. This covariance can be written as 



I(yii-yf')(y*-yr) 

c<jk> = UlLl i k = 1 2 D 



where is the number of cases observed for Yj and and yj and yj^ 
ajPe sample means calculated over N^^^. Let sjj^^and sj^^'^) be sample 
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variances of yj and yj^ for available cases, respectively. Then, an estimate 
of the jk^ PAC correlation can be expres 



I expressed as 



r = ^ 



(2) 



A small artificial example reported in Little and Rubin (1987, pg. 43) 
shows that when p > 3 the expression in (2) can give rise to non-positive 
definite correlation matrices and covariance matrices (see also Browne, 
1982, pg. 88). 

An additional problem associated with PAC covariance matrices is 
that they do not maximize any proper likelihood (Muthen, Kaplan, & 
HoUis, 1987). Specifically, as shown by Heiberger (1977, see also Brown, 
1983), the asymptotic covariance of PAC covariances in (1) can written 
as 



aCOV(S^,S^) (^jl^km^<^jm<ykl) 



jk^^hn (3) 



where Njy^^ is the number of complete observations on variables j, k, 1, 
and m. It can be seen that the multiplier (^jklm^^jk^lm) ^PP^^ for the 
maximum likelihood and generalized least squares discrepancy function 
instead of the usual UN. Moreover, as noted by Browne (1982), the 
discrepancy function formed from the analysis of PAC covariance matri- 
ces is not bounded by zero so that it is not a true discrepancy function. 
Thus, the Wishart distributional assumptions are violated and this will 
most probably affect the chi-square goodness-of-fit test (Bollen, 1989). 
One purpose of this paper is to study the affects of PAC covariance 
matrices on the empirical distribution of the chi-square test 
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3. An Alternative Approact) for Factor 
Analyzhg BIB Spiralled Data 



I 3.1 Theoreftcal Background 

I In this paper, a procedure is proposed that addresses the problem of 
missing data arising from BIB spiralling designs. The proposed method is 
based on recent developments in missing data theory developed by Rubin 
( 1976, see also Little and Rubin, 1987) and applied to structural equation 
I modeling by Muthen, Kaplan, and HoUis (1987). Muthen et al. consider 
the factor analysis model 



y* = v+Ar|+£ (4) 



where y* is a vector of potentially observable response variables, v is a 
vector of intercepts, A is a matrix: of factor loadings, n is a vector of 
common factors, and e is a vector of unique variables. 

In addition to the factor model in (4), Muthen et al. specify a vector of 
latent se* lection variables s* associated with each y* of the form 



s* ^r„n + Ty +5 . 



(5) 



I Associated with each s* variable is a threshold t; such that for the i^^ 
i observation. 



Jlifs:>T. 

Sj. = i 

[O, otherwise 
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In other words, if sjj = 1 then the unit's value on y*j is selected to be 
observed as yj and hence is not missing. If sjj = 0, then yy is missing. Thus 
in this framework, each s* variable determines if tlie corresponding y* 
variable is to be observed as yj, or is missing. The strength of the 
selectivity is determined by the elements in r while the amount of 
missingness is determined by t. 

Under the specification in (4) and (5), Muthen et al. derive the 
likelihood of the (not missing) observations on y*. Specifically, Muthen 
et al. consider two vectors of parameters: a vector for the factor analysis 
model in (4) and a vector for the selection model in (5). Under general 
assumptions, the stacked vector of parameters for the factor model can be 
written as 

(6) 



where V is the covariance matrix of x], and &^ is the diagonal matrix of 
unique variances. Analogously, the parameters for the selection equation 
in (5) can be stacked in the vector 



where 



i 
i 



■is 

1 



(}> = (r,05,05g,T), I 

(7) I 



where Ogg allows for the possibility that e and 5 are correlated. The | 
likelihood of y is a function of 9 and (|> and can be written as (see Rubin, | 
1976; Little, 1982, 1983; Muthen et al, 1987) I 



log L(e,4)|y)= flog(}>'(0|y) + log/(e,(J>|y). | 

(8) i 



log (|)*(e|y) = const. - - log|E* , 



i 



(9) I 
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I The expression in (8) is referred to as the '*true likdihood" (Muthen et al„ 

I 1987). The first term on the right hand side of (8) is the multivariate normal 

I likelihood for the /VS sample units displaying the pattern of missing 

I data, and is referred to as the **quasi-Iikelihood'' since it ignores the 

I missing data generating mechanism. The second term on the rhs of (8) 

I represents the mechanism that generates the missing data. 

I 

I Correct approaches to the maximization of (8) depend on whether the 

I missing data mechanism, log/6,(|)|y), is '^ignorabie'' or "non-ignorable.'' 

i In this paper we will consider the use of the quasi-likelihood estimator 

I under one restrictive condition of "ignorability" of the missing data 

I mechanism, namely that the missing data are missing completely at 

I random (MCAR). Conditions for MCAR are = 0, Fy = 0 and 05. = 0, 

i Although MCAR is a restrictive and usually unreasonaole assumption for 

j most situations it is the correct assumption for properly implemented BIB 

I designs (Zwick, 1987). Under the assumption of MCAR maximization of 

I the quasi-likeJihood with respect to 6 yields conect estimates of model 

I parameters, correct chi-squares and conrect standard errors. Muthen, et ai 

I (1987) refer to this estimator as the full quasi-likeUhood (FQL) estimator. 

i 

I The FQL estimator was compared to hstwise and pairwise approaches 

I in an extensive population study by Muthen et al, (1987). A variety of 

I conditions were examined, including the less restrictive condition of 

I missing-at-random (MAR; Little & Rubin. 1987) wherein ignorability did 

I not hold The general findings were that the FQL estimator was superior 

I to traditional approaches even under cases where it was not correct to 

I ignore the right hand side of (8). Missing from the Muthen et al. study was 

I a comparison of these approaches with respect to the empirical distribu- 

I tion of the chi-square goodness-of^fit test. 

I 3.2 Software Implemenfatlon of FQL Estimator 

I Implementation of the FQL estimator for a single population with g 

I distinct missing data patterns makes use of the multiple group factor 

j analysis approach available in LISREL (Joreskog, 1989), LISCOMP 

I (Muthen, 1987), and HQS (Binder, 1989). With respect to this paper, the 

I groups correspond to the booklets. The procedure requires two steps. In 

I the first step, the alternative hypothesis (H^) of a pooled unrestricted 

I covariance matrix is specified. This requires formation of g covariance 

I matrices corresponding to g booklets but based on die item responses to 

I all the tasks. By design, these correlation matrices will have missing data 

I corresponding to blocks of items not represented in booklets. The vari- 

I ances and covariances corresponding to missing data in each booklet are 

% coded one and zero, respectively. Once the matrices are formed, identifi- 

I able variance and covariance elements common to each booklet are 
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constrained equal. It may be interesting to note that the chi-square test | 
associated with this step tests whether the missing data are MCAR. | 

In the second step, the null hypothesis (Hq) corresponding to the k- 1 

factor model is specified. Utilizing the same multiple group setup as in the | 

first step, the model imposes equality con straints among the common and | 

identifiable parameters of the factor model. A large sample chi-square test | 

of the null hypothesis of /t factors is carried out by subtracting the H [ chi- S 

square from the Hq chi-square with degrees-of^freedom equal to the | 

difference in corresponding degrees-o^-freedom, | 

It should be noted that for the purposes of this paper we are assuming | 
there is no additional missing data apart from that induced by the BIB 
design. Clearly, this is not a reasonable assumption in practice since 
omitted responses can occur for administered items. The method for 
handling omits in NAEP Young Adult Literacy Survey, for example, was 
to treat omits after the last valid response in a block as not-reached and 
omits prior to the last valid response in a block as missing. 



1 
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4. Design of Simulation Study 



In order to compare the FQL procedure to the standard approach of 
analyzing PAC covariance matrices, a simulation study was employed. 
Two cases were studied: Case 1 consisted of continuous variables in order 
to test the new approach under ideal conditions. Case 2 consisted of 
dichotomous variables in order to test the new approach under more 
realistic conditions of item level data. Case 2 studies the problem of factor 
analyzing PAC phi correlation matrices and represents the approach take 
by Kirsch and Jungeblut (1986) in the analysis of the Young Adult 
i Literacy Survey. 

! 

I For both cases, one hundred replications of 1 000 random variates were 
generated from a multivariate normal distribution with a covariance 
matrix that followed a nine- variable/one-factor model. All factoi* loadings 
were specified to be 0.7 in the population. This model possesses 27 
degrees-of-freedom. For each replication, the data matrix was modified 
according to a BIB spiralled design in a manner similar to Zwick (1987). 
That is, the first three items formed Block A, the next three items formed 
Block B, and the last three items formed Block C. The first 333 examinees 
received Blocks A and B, the next 333 examinees received Blocks B and 
C, and last 334 examinees received Blocks A and C. Three BIB spiralled 
covariance matrices were formed in SPSS (SPSS Inc, 1989) and used as 
input to the multiple group specification of LISCOMP (Muthen, 1987). In 
addition, PAC covariance matrices were also formed using SPSS for each 



I replication and analyzed via LISCOMP. For the PAC case two sample 
I sizes were chosen: an average sample size of 466 and a lower bound 



I sample size of 333. 



m 
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5. Results 



10 occAsiONAiPArae om*i 
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Recent theory given in Muthen, et al, (1987) show that under MCAR 
the PAC approach will yield no large sample bias. A cursory inspection 
of the results of this study show this to be the case as well. Thus, for ease 
of presentation, results are presented as they pertain to the empirical 
distribution of the chi-square goodness-of-fit test. Results are presented in 
terms of the mean and variance of the chi-square statistic over 100 
replications as well as the rejection frequency for nominal Type I error 
levels of .05, -01, and .001- 

5 J QualHy of Data Genercdion 

As a check on the quality of the data generation process, complete data 
were studied. Recalling that the df = 27, the results show that across 100 
replications, the mean = 25.965, the variance of = 43.470. The 
rejection frequencies for nominal Type I errors of .05, .01, and .001 were 
2, 1 , and 1 respectively. Thus, we find that there is a slight underestimation 
of the expected values for the chi-square distribution with 27 df. While | 
these values should be kept in mind when examining the results below, it | 
was decided that 100 replications was adequate to gauge the behavior of | 
the chi-square distribution for this study. | 

5 J Case 1: Continuous Variables I 

i 

The upper panel of Table 1 presents results comparing the FQL | 

estimator to the standard approach of analyzing PAC covariance matrices | 

under ideal conditions of continuous normally distributed variables. It can | 

be seen that the PAC approach yields clearly unacceptable rejection I 

frequencies for both conditions of sample size. It should be noted that all || 

replications converged properly, thus the results can only be explained by p 

the violation of Wishard distribution assumptions induced by PAC cova- | 

nance matrices. By comparison, the FQL estimator shows nearly perfect | 

performance with respect to mean, variance and rejection frequency of the | 

chi-square distribution. | 

5.2 Case 2: Dichotomous Variables - Phi Coefficients 

In order to generate dichotomous variables, the continuous variables 
from the complete data were dichotomized on the basis of a 70/30 split 
This corresponds to a threshold k, say, of .52. All variables were dichoto- 
mized according to this threshold so as not to confound the possible effects 
of PAC data with the occurrence of difficulty factors arising from unequal 



1 



i 



thresholds in the population. BIB spiralled data were then created in the 
manner described in Section 4. 

The results for the analysis of phi coefficients are displayed in the 
lower panel of Table 1. It can be seen that for dichotomous data the PAC 
approach yields unacceptable rejection frequencies and are wone than 
those found for the continuous data in Section 5,2. This result can be 
explained by noting that chi-square goodness-of-fit tests are sensitive to 
categorization (see e.g., Boomsma, 1983; Olsson, 1979) Here, the effects 
of categorization interact with distributional violations induced by PAC 
created correlation matrices. Ey comparison the FQL estimator performs 
better but rejection frequencies are still unacceptable. 



I Table L 

I Chi-square mean, variance, and rejection frequencies for PAC and FQL 
I approaches on continuous and dichotomous variables^ 



i 



Estimator 

ML 



Data Generation Check 

Rejection Frequencies 

Meon Variance .05 .01 .001 

25.965 43.470 2 1 1 



I 



i 



Continuous Variables 

Rejection Frequencies 



Estimator 


Mean 


Variance 


.05 


.01 


.001 


PAC-466t» 


50.404 


274.497 


68 


53 


36 


PAC-333C 


35.944 


139.592 


33 


19 


08 


FQL 


28.034 


53.548 


05 


00 


00 






Dichotomous Variables 






EsKmator 


Mean 


Voriance 


.05 


.01 


.001 


PAC-466 


56.575 


276.532 


87 


69 


50 


PAC-333 


40.622 


139.278 


47 


28 


13 


FQL 


28.034 


53.548 


35 


14 


05 



I a E(x^) = 27,Var(x^)=54 

■|| ^ Based on average sample size of 466. 

|| ^ Basedonminimuinsamplesizeof333. 
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This papei* was motivated by the problem of assessing the literacy 1 

skills of young adults when the data are obtained from a BIB spiraling | 

scheme. The results of this study suggest that caution must be exercised | 

when interpreting goodness-of-fit tests from data arising from BIB spi- | 

railed designs using PAC approaches. Specifically, this paper demon- j 

strates that the chi-square goodness-of-fit test is extremely sensitive to I 

violations of Wishart distribution assumptions arising from the construe- | 

tion of PAC covarlance matrices even when the missing data are MCAR. | 

Recall that even in the ideal case of continuous variables, 68% of the i 

models were rejected at the 5% level. While these results may explain 1 

Zwick's (1987) finding of a large goodness-of^fit value for her simulated ^ 

BIB data, she is essentially correct in stating that the random nature of the i 

spiralling procedure does not guarantee that dimensionality is unaffected J 

This paper provides the statistical justification for why such problems J 

might occur. | 



I 

I 



si 



i 

m 



Of particular note is the behavior of the FQL estimator proposed by 
Muthen et al. (1987) . The results indicate extremely good performance for 
continuous variables, and relatively better performance for dichotomous 
variables. Nevertheless, the performance of the FQL estimator for di- 
chotomous variables was not adequate and more extensive theory for this 
case needs to be developed. In particular, the approach of Muthen et al. 
(1987) must be extended to include more general estimation procedures 
which explicitly take into account the categorical nature of item nesponses 
(e.g.. Bock, Gibbons, & Muraki, 1985; Muthen, 1984). | 

As in all simulation studies, generalizability is limited by the choice 1 
of levels of independent variables. The purpose of this paper was to focus § 
on a partiicular pattern of missing data and to provide an alternative to | 
assessing goodness-of^fit A factor not examined m this study was the I 
sample size ratios. It may be the case that PAC approaches perform | 
relatively better when less extreme sample size ratios are found However, || 
given that MCAR is unrealistic except for perhaps aU but BIB designs, and 
given tiiat tiie sample size ratios for this study are representative of those 
found in BIB designs such as the Young Adult Literacy Survey, it is felt 
that these results speak to the problems of ignoring the missing data 
pattern when factor analyzing data from BIB spiralled assessments. 



A final consideration is the problem of omitted responses not induced | 
by tile BIB design. A standard approach for dealing with tiie problem i 
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omitted responses for such assessments as the NAEP Adult Literacy study 
is to treat omitted items after the last valid response within a block as not- 
reached and omitted responses before the last valid response as incorrect 
From the standpoint of item response theory this may be a sensible 
approach (see Lord, 1980). However, it appears that the argument for such 
an approach does not rest on a formal model which specifies whether 
omitted responses are omitted completely at random or are functions of 
other omissions where perhaps ignorability is not an appropriate assump- 
tion. It may be necessary for practical purposes however, to adopt this 
approach because application of the FQL estimator for general patterns of 
omitted responses would require covariance matrices to be formed for 
I every distinct missing data pattern. These resulting covariance matrices 
I would likely be based on unreasonably small sample sizes. A similar 
I concern was raised by Muthen et al. (1987) regarding the application of 
^ the FQL estimator for general structural equation models. Nevertheless, 
I if cunent approaches to handling omits and noH^eached responses can be 
I justified, then the FQL estimator appears to be a superior approach for 
treating BTB induced missing data patterns. 
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